Exploring Humanistic Intelligence Through Physiologically Mediated Reality

We present a way of making the wearing of a lifelong electrocardiographic health monitor fun for a user. The health monitor is coupled with a reality mediator device to create a Physiologically Mediated Reality. Physiologically Mediated Reality is Mediated Reality which alters a user’s audiovisual perception of the world based upon their own electrocardiographic waveform. This creates an interesting audiovisual experience for the user, playing upon the poetic narrative of combining cardiocentric metaphors pervasive in everyday life (the heart as a symbol of love and centrality, e.g. “get to the heart of the matter”) with the ubiquitous occularcentric metaphors such as “see the world from my point of view”. This audiovisual experience is further enhanced by combining music which alters the visual perception and also heightens the user’s emotional response to their experience, and in doing so, further affects their heart(beat). 1 Ambulatory Physiologically Mediated Reality In the same way that the bars of a simple spectrum analyzer rise and fall along with along with the beat (or, more accurately, the frequency spectrum of the music), we have developed a Physiologically Mediated Reality in which a video stream of what the user sees is altered in time with their own heartbeat. Of interest is the synthesis of cardiocentric and occularcentric metaphors common to society. Since the heart is often linked with emotional response (we are often said that we “feel” with our hearts), the emotional experience of a physiologically mediated reality was coupled with an auditory mediated reality, where music was used to create a heightened emotional response to the experience (see figure 1). Through the use of EyeTap (a device which allows the eye of a wearer to function both as a camera and as a display) [5], the field of view of the wearer is altered in time with both the music and wearer’s heartbeat (see figure 2). ECG EyeTap AV Figure 1: Signal flow paths in Physiologically Mediated Audiovisual (AV) Reality Figure 2: Screenshot of computationally altered live video in which the the environment appears to swirl and pulse to the beat of the wearer’s heart. We combined our EyeTap system with common wearable personal sound systems. Our EyeTap eyeglasses based implementation currently allows the video to “saturate” between beats (i.e. video frames are superimposed on top of one another), and then clear to a fresh video frame on either a heart beat or a musical beat. Musical bass beats were detected by analyzing the spectrum analyzer output from software music player. The real time video saturation effect was achieved by applying the X–windows GXand and GXor functions to the updated video. It was found that this system created the impression of visual reality cardiographically and musically pulsating along with the beat of the heart, allowing the music to be experienced both visually and aurally. The physiological control system allowed the user’s emotional response to further affect the system, and this, in turn, further affected their experience and emotional response, thus creating a closed loop feedback system. Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR’02) 0-7695-1781-1/02 $17.00 © 2002 IEEE